Under Review in Neural Computation , 2002 Comparison of Model Selection for Regression

We discuss empirical comparison of analytical methods for model selection. Currently, there is no consensus on the ‘best’ method for finite-sample estimation problems, even for the simple case of linear estimators. This paper presents empirical comparisons between classical statistical methods (AIC, BIC) and the SRM method (based on VC-theory) for regression problems. Our study is motivated by empirical comparisons in (Hastie et al, 2001) who claim that SRM method performs poorly for model selection. Hence, we present empirical comparisons for various data sets and different types of estimators (linear, subset selection and k-nearest neighbor regression). Our results demonstrate practical advantages of VC-based model selection, as it consistently outperforms AIC and BIC for most data sets (including those used in (Hastie et al, 2001)). This discrepancy (between empirical results obtained using the same data) is caused by methodological drawbacks in (Hastie et al, 2001)) especially in their loose interpretation and application of the SRM method. Hence we discuss methodological issues important for meaningful